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Background / Context: 

As described in the background for Paper 1, the consequences of failing core academic courses 
during the first year are dire. In Chicago, over a quarter of students fail at least one semester of 
algebra in their ninth grade year, and only 13% of students who fail both semesters of Algebra I 
in ninth grade graduate in 4 years. 

Offering credit recovery options is one strategy to deal with high failure rates. The primary goal 
of credit recovery programs is to give students an opportunity to retake classes that they failed in 
an effort to get them back on track and keep them in school (Watson & Gemin, 2008). It makes 
theoretical sense to try to get students to recover their algebra credits early, in the summer after 
ninth grade — before they take geometry or Algebra II and chemistry or physics, which may 
require mastery of Algebra content matter. Recovering credits earlier may also put students back 
on track towards accruing the credits necessary to progress to the next grade and ultimately 
graduate. However, offering summer classes takes resources — time, money, and coordination. 
Schools often concentrate their credit recovery efforts on students that are close to graduation, 
rather than ninth graders. In fact, there is little evidence about the extent to which getting 
students to recover credits early leads to substantive improvements in student progression and 
later outcomes. While it seems like a good idea, the pay-off may not actually be large for a 
number of reasons: few students who failed in the prior year may show up in the summer for 
credit recovery; few students may pass even if they do show up; and the gains of attending 
summer school for learning and for credit accumulation may be very small compared to students’ 
initial deficits in skills or the number of total credits they eventually need to recover. Thus, 
schools might put in substantial effort to hire teachers and find facilities for credit recovery with 
little pay-off in terms of their schools’ on-track rates, test scores, and later graduation rates. 

Purpose / Objective / Research Question / Focus of Study: 

This study examines the benefits of offering expanded credit recovery options for ninth grade 
algebra, relative to business as usual (i.e., the summer programming schools would offer in the 
absence of efforts to expand credit recovery). Evidence of the effects of credit recovery for 
getting students back on track for graduation is lacking, and this study presents the opportunity to 
generate such evidence. 

Setting: 

The setting will be neighborhood high schools in Chicago in 2009-2011. CPS is the third-largest 
U.S. district. The district is 87% low-income and 42% African American and 44% Latino. Eor 
this study, we include all neighborhood high schools (no charter or selective enrollment schools) 
that had incoming ninth grade cohorts for all four years, excluding a group of schools that were 
participating in another district-sponsored algebra program. 
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Population / Participants / Subjects: 

The population consists of all first-time ninth grade students who entered regular neighborhood 
high schools between fall 2009 and fall 2011 who failed second semester algebra (Algebra IB). 
There were about 4,000 students in each cohort. All neighborhood high schools were included 
for analysis (76 schools). 

Intervention / Program / Practice: 

In this study, some high schools in CPS were given the resources to offer second-semester 
Algebra I credit recovery courses to as many ninth grade students as they could recruit. In 
addition, they received support for recruiting students who failed Algebra IB to come to summer 
school, as well as logistical support around finding qualified teachers. Schools received funding 
to implement at least two Algebra I credit recovery courses during the summer sessions of 201 1 
and 2012 — at least one online and one face-to-face section; some schools offered four or six 
sections of algebra credit recovery, which were paid for by the program. 

Not all schools were invited to participate in the study. Schools were invited to participate in the 
study based on two main criteria. The primary criterion was the number of students who failed 
second semester algebra in 2010. Researchers ranked schools by the number of students who 
failed in the year prior to the start of the study (2010) and considered those with the most failures 
eligible to be invited to participate. Those above the cut-off for eligibility were invited, if they 
met the other criterion, while those below the cut-off were not. Schools could have large 
numbers of algebra failures because they had high failure rates, or because they were large 
schools, or both. The second criterion was that they were not closed for construction over the 
summer, and could potentially offer summer school. Each summer, a number of schools were 
closed for construction. In addition, we excluded charter schools from consideration because 
their students’ transcripts do not appear in the centralized data system. We also did not consider 
schools that were in one area of the district whose area chief had refused participation because of 
other algebra-related initiatives being implemented. 

Fifteen schools participated in 201 1; in total they offered 18 pairs of sections (36 total). Thirteen 
schools participated in 2012; in total they offered 20 pairs of sections (40 total) (see paper 1 for 
details). The participating schools were similar to other schools in CPS in most respects, with a 
few notable differences. They were larger schools than typical; they enrolled 1,785 students, on 
average, compared with an average enrollment of 729 students in high schools across CPS. This 
difference was expected because we could implement the study only in schools that were large 
enough to have sufficient students to fill at least two sections of second-semester algebra over 
the summer. The study schools also disproportionately served more Latino students and fewer 
African-American students than the district as a whole. This occurred because large schools are 
more likely to serve Latino students while small schools are more likely to serve African- 
American students. 

The extra funding schools received as part of participating in this study increased their capacity 
to provide algebra credit recovery over the summer so that all ninth grade students who wanted 
to recover the credit over the summer could do so. To boost enrollments in summer algebra 
credit recovery, a set of outreach activities were implemented in each school to encourage ninth 
grade students who failed the second semester of Algebra I to attempt to recover the credit in the 
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summer. This included presentations to groups of failing students by study team members, calls 
to students’ homes, and letters sent home about the importance of passing algebra and recovering 
credits. We define the presence of the funding for extra courses, plus the push to motivate 
students to enroll in credit recovery, as “expanded credit recovery options.” 

Research Design, Data Collection and Analysis: 

This study conducts two types of analysis, using difference-in-difference approach where there 
are multiple embedded comparisons. We model outcome differences between two pre- 
intervention cohorts (2009 and 2010) and two post-intervention cohorts (2011 and 2012) in 
schools that were offered the opportunity to participate in expanded credit recovery and those 
that were not. The outcome variables include recovery rates in Algebra IB, promotion to 10* 
grade rates, 10* grade PLAN scores on the Pre- Algebra/ Algebra subtest, as well as course taking 
and course performance in the 10* grade year. 

We perform two school-level analyses and one student-level analysis. The first school-level 
analysis examines the intent-to-treat (ITT) effect— the effect of receiving an invitation to 
participate in expanded summer credit recovery. This analysis exploits the fact that invitation 
was determined rigidly by a known characteristic— the number of failures in 2010. We compare 
differences in student outcomes from pre- to post-intervention years within schools, and the size 
of these differences between schools eligible and not eligible for participation. We include 
failure rates and school size in the current year as control variables. Additional control variables 
include students’ background characteristics— students’ math scores on standardized tests at the 
end of eighth grade and beginning of ninth grade, number of semester courses failed in ninth 
grade, whether they failed 0, 1 or two semesters of algebra, race, gender, SES, year fixed effects 
and school fixed effects. 

The second school-level analysis examines the treatment-on- the-treated (TOT) effect, that is, the 
effect of accepting an invitation to participate in expanded summer credit recovery. This analysis 
exploits the fact that invitation was determined by the number of failures in 2010 using an 
instrumental variables framework. We assume that, conditional on cohort size and failure rates in 
the current year, the number of failures in a past year is only related to our outcome of interest 
through participating in expanded credit recovery. Thus, we use eligibility status in a year to 
instrument for participation. We include failure rates and school size in the current year as 
control variables in both stages of the analysis. Additional control variables include students’ 
background characteristics— students’ math scores on standardized tests at the end of eighth grade 
and beginning of ninth grade, number of semester courses failed in ninth grade, whether they 
failed 0, 1 or two semesters of algebra, race, gender, SES, year fixed effects and school fixed 
effects. 

The student-level analysis examines the effect of successfully recovering an algebra credit over 
the summer using an instrumental variables approach. Appealing to the same logic as above, we 
use school-by-year eligibility to participate in expanded credit recovery to estimate the compliers 
average treatment effects of recovering a credit. We take advantage of the two known exogenous 
sources of variation affecting attendance in Algebra IB in the summer. One source of variation 
was whether schools received an invitation to participate on the basis of the number of failures in 
2010. A second source of variation was whether or not the school was under construction, which 
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was decided by the district independently from this study. We use additional control variables as 
described above. 

Findings / Results: 

We find that that offering resources to schools to expand credit recovery for ninth graders 
substantially increases their ninth grade recovery rates. The effect of being offered a chance to 
participate (ITT) increases recovery rates from 12% to 20% on average. The effect of 
implementing the expanded recovery options doubles recovery rates from 13% to 27%. While 
the recovery rate doubled, the percentage of students who recover is still low, largely because 
only a subset of students who fail show up for recovery, even when sufficient resources are 
available. In addition, many schools chose not to expand credit recovery, even though they 
would receive additional resources; approximately 55% of schools that were offered the 
opportunity to participate accepted. Despite these notable changes in recovery rates, preliminary 
models suggest that schools do not see a detectable benefit from participating in terms of test 
scores, course-taking, and course performance in students’ second year of high school. However, 
while not statistically significant, there were differences by treatment in whether students’ 
records in their second year of high school were observed, and final conclusions about second 
year outcomes will require missing data models that are currently being developed. In addition, 
the act of receiving credit itself may have contributed directly to students’ probability of staying 
in school and graduating, which is an outcome that is not yet observable. 

Preliminary analysis of the student-level models indicates that recovering credits over the 
summer impacts promotion to 10* grade, whether or not a student takes the PLAN exam, and the 
level of difficulty of course selection in the next school year. By imputing student-level missing 
data (when missing due to not being promoted to 10* grade etc.), we may be able to estimate the 
ways in which observable later outcomes are driven by sample selection due to credit recovery 
and produce reasonable estimates as to the true local effect of credit recovery. Should the data 
permit, we will also look for heterogeneous effects by the number of courses failed in 9* grade. 
We will examine if pay-offs to credit recovery differ by the size of the initial credit deficit. 

Conclusions: 

While expanding credit recovery options may seem like an attractive strategy to improve future 
student outcomes and move students further on the path to graduation, the estimated pay-offs 
relative to the costs may imply that expanding summer credit recovery is not a prudent use of 
resources. While recovery rates increase with expanded recovery options, most students still do 
not recover, and schools do not see a definitive increase even in scale with the students who do 
recover. This study is not yet able to identify impacts on graduation (4-year graduation for 
students in the second cohort will take place June 2015), a key outcome of interest, and early 
credit recovery may be more likely to affect graduation than second year performance. Pending 
forthcoming results, we will discuss the possibility of heterogeneous effects by initial credit 
deficit. 
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Appendix B. Tables and Figures 


Figure 1. Theory of Action Behind Summer Online Algebra Credit Recovery 



Table 1. Characteristics of CPS High Schools Participating in Credit Recovery Study in 


Summer 2011 and District High Schools Overall, as of 2010 



2011 Study Schools 

All CPS high 
schools 


Average 

Average 

Average 

Characteristics 

Number 

Percent 

Percent 

Female 

906 

50.7% 

49.6% 

Race/Ethnicity 




White 

136 

6.0% 

4.7% 

African American 

628 

42.4% 

59.4% 

Hispanic 

957 

48.0% 

32.5% 

Asian 

4 

0.2% 

0.2% 

Native American 

8 

0.4% 

0.2% 

Other Race 

17 

0.9% 

1.2% 

Eligible for free or reduced-price lunch 

1468 

91.5% 

91.3% 

Home language not English 

975 

48.3% 

31.2% 

Eligible for special education services 

269 

15.4% 

18.9% 


Number of study schools is 15. Averages are calculated from all students in grades 9-12 active during the fall semester, 2010. 
District averages include all schools with students in grades 9-12 (total school N=150). 
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Table 2. Characteristics of CPS High Schools Participating in Credit Recovery Study 


During Summer 2012 and District High Schools Overall, as of 2011 



2012 Study Schools 

All CPS high 
schools 

Characteristics 

Average 

Number 

Average 

Percent 

Average 

Percent 

Female 

907 

49.6% 

49.4% 

Race/Ethnicity 




White 

179 

8.2% 

4.7% 

African American 

497 

33.3% 

58.4% 

Hispanic 

1076 

54.5% 

33.1% 

Asian 

2 

0.1% 

0.2% 

Native American 

8 

0.4% 

0.2% 

Other Race 

22 

1.4% 

1.3% 

Eligible for free or reduced-price lunch 

1572 

91.0% 

91.6% 

Home language not English 

1074 

53.4% 

31.4% 

Eligible for special education services 

287 

16.0% 

19.2% 


Number of study schools is 13. Averages are calculated from all students in grades 9-12 active during the fall semester, 2011. 
District averages include all schools with students in grades 9-12 (total school N=150). 


Table 3. Number and Percentage of Students Per Condition by Block, Summer 2011 




Passed Algebra lA 

Failed Algebra lA 

Algebra lA Status 
Unknown 

Total 

Condition 

Gender 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 


F2F 

Female 

44 

15% 

28 

10% 

31 

11% 

103 


Male 

70 

24% 

58 

20% 

59 

20% 

187 


Total 

114 

39% 

86 

30% 

90 

31% 

290 

Online 

Female 

45 

15% 

37 

12% 

35 

11% 

117 


Male 

73 

24% 

60 

20% 

56 

18% 

189 


Total 

118 

39% 

97 

32% 

91 

30% 

306 
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Table 4. Number and Percentage of Students Per Condition by Block, Summer 2012 




Passed Algebra lA 

Failed Algebra lA 

Algebra lA Status 
Unknown 

Total 

Condition 

Gender 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 


F2F 

Female 

56 

14% 

52 

13% 

41 

10% 

149 


Male 

83 

21% 

95 

24% 

70 

18% 

248 


Total 

139 

35% 

147 

37% 

111 

28% 

397 

Online 

Female 

53 

13% 

55 

14% 

44 

11% 

152 


Male 

81 

21% 

93 

24% 

69 

17% 

243 


Total 

134 

34% 

148 

37% 

113 

29% 

395 


Table 5. Baseline Characteristics of Cohort 1 (Summer 2011) 


Characteristic 

Online 

F2F 

p-value 

Mean spring 2010 Explore math scaled score 

13.45 

(2.92) 

13.25 

(2.96) 

0.193 

Mean concentrated poverty (2009 ACS) “ 

0.13 

(0.75) 

0.12 

(0.74) 

0.912 

Mean social status (2009 ACS) 

-0.57 

(0.87) 

-0.54 

(0.85) 

0.743 

Mean number of unexcused absences 
(2010-2011 school year) 

32.05 

(23.48) 

30.49 

(23.32) 

0.289 

Percent first-time freshman 

88 

91 

0.194 

Percent special education 

10 

7 

0.216 

Percent African American 

38 

35 

0.226 

Percent Latino 

56 

59 

0.253 

Percent Other Race (non-Latino, non-African 
American) 

6 

6 

0.821 

Percent Suspended 
(2010-2011 school year) 

46 

46 

0.830 

Percent Moved Schools 
(2010-2011 school year) 

5 

5 

0.801 

Percent Female (blocking variable) 

38 

36 

0.629 

Percent Passed Algebra lA (blocking variable) 

39 

40 

0.574 

Percent Failed Algebra lA (blocking variable) 

32 

30 

0.575 

Percent Unknown Pass/Fail in Algebra lA (blocking 
variable) 

30 

30 

0.989 


Note: Sample includes 15 schools; 591 students (304 Online, 287 F2F). Values represent unadjusted means. Differences in 
characteristics by condition were tested using a model that modeled schools and summer school session as fixed effect to 
account for the clustering of students within schools and summer school session. Figures in parentheses are standard deviations. 

a. Concentration of poverty is a standardized measure of poverty for the census block group in which the student lives. A large 
positive number indicates a high level of poverty concentration; a large negative numbers indicates a low level of poverty 
concentration. This measure is calculated from Census data (the percent of adult males employed and the percent of families 
with incomes above the poverty line), and is standardized such that a “0” value is the mean value for census block groups in 
Chicago. 

b. Social status is a standardized measure of educational attainment/employment status for the census block group in which the 
student lives. A large positive number indicates a high social status; a large negative numbers indicates a low social status. 

This measure is calculated from Census data (mean level of education of adults and the percentage of employed persons who 
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work as managers or professionals), and is standardized such that a “0” value is the mean value for census block groups in 
Chicago. 

Source: Chicago Public Schools (CPS) Administrative Data 


Table 6. Baseline Characteristics of Cohort 2 (Summer 2012) 


Characteristic 

Online 

F2F 

p-value 

Mean spring 2011 Explore math scaled score 

13.64 

(2.83) 

13.78 

(2.88) 

0.354 

Mean concentrated poverty (2009 ACS) “ 

-0.03 

(0.79) 

0.01 

(0.76) 

0.574 

Mean social status (2009 ACS) 

-0.40 

(0.86) 

-0.45 

(0.87) 

0.475 

Mean number of unexcused absences 
(2011-2012 school year) 

24.03 

(20.85) 

25.86 

(21.51) 

0.246 

Percent first-time freshman 

87 

88 

0.586 

Percent special education 

9 

10 

0.521 

Percent African American 

31 

28 

0.107 

Percent Latino 

58 

59 

0.533 

Percent Other Race (non-Latino, non-African 
American) 

12 

13 

0.511 

Percent Suspended 
(2011-2012 school year) 

34 

37 

0.391 

Percent Moved Schools 
(2011-2012 school year) 

5 

6 

0.437 

Percent Female (blocking variable) 

39 

38 

0.740 

Percent Passed Algebra lA (blocking variable) 

34 

35 

0.688 

Percent Failed Algebra lA (blocking variable) 

38 

37 

0.790 

Percent Unknown Pass/Fail in Algebra lA (blocking 
variable) 

29 

28 

0.868 


Note: Sample includes 13 schools; 792 students (395 Online, 397 F2F). Values represent unadjusted means. Differences in 
characteristics by condition were tested using a model that modeled schools and summer school session as fixed effect to 
account for the clustering of students within schools and summer school session. Figures in parentheses are standard deviations. 

a. Concentration of poverty is a standardized measure of poverty for the census block group in which the student lives. A large 
positive number indicates a high level of poverty concentration; a large negative numbers indicates a low level of poverty 
concentration. This measure is calculated from Census data (the percent of adult males employed and the percent of families 
with incomes above the poverty line), and is standardized such that a “0” value is the mean value for census block groups in 
Chicago. 

b. Social status is a standardized measure of educational attainment/employment status for the census block group in which the 
student lives. A large positive number indicates a high social status; a large negative numbers indicates a low social status. 

This measure is calculated from Census data (mean level of education of adults and the percentage of employed persons who 
work as managers or professionals), and is standardized such that a “0” value is the mean value for census block groups in 
Chicago. 

Source: Chicago Public Schools (CPS) Administrative Data 
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